Transferable Post-hoc Calibration on Pretrained Transformers in Noisy Text Classification

نویسندگان

چکیده

Recent work has demonstrated that pretrained transformers are overconfident in text classification tasks, which can be calibrated by the famous post-hoc calibration method temperature scaling (TS). Character or word spelling mistakes frequently encountered real applications and greatly threaten transformer model safety. Research on under noisy settings is rare, we focus this direction. Based a toy experiment, discover TS performs poorly when datasets perturbed slight noise, such as swapping characters, results distribution shift. We further utilize two metrics, predictive uncertainty maximum mean discrepancy (MMD), to measure shift between clean datasets, based propose simple yet effective transferable for calibrating models dynamically. To evaluate performance of proposed methods settings, construct benchmark consisting four noise types five intensities QNLI, AG-News, Emotion tasks. Experimental show (1) metrics measuring (2) significantly decrease expected error (ECE) compared with competitive baseline ensemble approximately 46.09%.

برای دانلود باید عضویت طلایی داشته باشید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Inductive Voltage Transformers Calibration by the Parameters

The accuracy class of an IVT – Inductive Voltage Transformer – is typically assessed in laboratory installations either by comparing with another IVT presenting greater accuracy and traceable to a national laboratory or by using a capacitive divider. Calibration in the field using internal parameters is considered herein, using results obtained from typical open and short circuit tests and wind...

متن کامل

From Text to Hypertext: A Post-Hoc Rationalisation of LaTeX2HTML

The conventional hypertext authoring framework compels authors to represent their material as an interconnected network of nodes and links. Apart from the diiculties that this alone entails, the situation with HTML is even more problematic since the author is also responsible for mapping the abstract network model onto the computer le system. This is likely to hinder the widespread adoption of ...

متن کامل

Cough event classification by pretrained deep neural network

BACKGROUND Cough is an essential symptom in respiratory diseases. In the measurement of cough severity, an accurate and objective cough monitor is expected by respiratory disease society. This paper aims to introduce a better performed algorithm, pretrained deep neural network (DNN), to the cough classification problem, which is a key step in the cough monitor. METHOD The deep neural network ...

متن کامل

Audio-based Music Classification with a Pretrained Convolutional Network

Recently the ‘Million Song Dataset’, containing audio features and metadata for one million songs, was made available. In this paper, we build a convolutional network that is then trained to perform artist recognition, genre recognition and key detection. The network is tailored to summarize the audio features over musically significant timescales. It is infeasible to train the network on all a...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

ژورنال

عنوان ژورنال: Proceedings of the ... AAAI Conference on Artificial Intelligence

سال: 2023

ISSN: ['2159-5399', '2374-3468']

DOI: https://doi.org/10.1609/aaai.v37i11.26632